6 research outputs found
Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization
Adversarial contrastive learning (ACL) is a technique that enhances standard
contrastive learning (SCL) by incorporating adversarial data to learn a robust
representation that can withstand adversarial attacks and common corruptions
without requiring costly annotations. To improve transferability, the existing
work introduced the standard invariant regularization (SIR) to impose
style-independence property to SCL, which can exempt the impact of nuisance
style factors in the standard representation. However, it is unclear how the
style-independence property benefits ACL-learned robust representations. In
this paper, we leverage the technique of causal reasoning to interpret the ACL
and propose adversarial invariant regularization (AIR) to enforce independence
from style factors. We regulate the ACL using both SIR and AIR to output the
robust representation. Theoretically, we show that AIR implicitly encourages
the representational distance between different views of natural data and their
adversarial variants to be independent of style factors. Empirically, our
experimental results show that invariant regularization significantly improves
the performance of state-of-the-art ACL methods in terms of both standard
generalization and robustness on downstream tasks. To the best of our
knowledge, we are the first to apply causal reasoning to interpret ACL and
develop AIR for enhancing ACL-learned robust representations. Our source code
is at https://github.com/GodXuxilie/Enhancing_ACL_via_AIR.Comment: NeurIPS 202
NoiLIn: Improving Adversarial Training and Correcting Stereotype of Noisy Labels
Adversarial training (AT) formulated as the minimax optimization problem can
effectively enhance the model's robustness against adversarial attacks. The
existing AT methods mainly focused on manipulating the inner maximization for
generating quality adversarial variants or manipulating the outer minimization
for designing effective learning objectives. However, empirical results of AT
always exhibit the robustness at odds with accuracy and the existence of the
cross-over mixture problem, which motivates us to study some label randomness
for benefiting the AT. First, we thoroughly investigate noisy labels (NLs)
injection into AT's inner maximization and outer minimization, respectively and
obtain the observations on when NL injection benefits AT. Second, based on the
observations, we propose a simple but effective method -- NoiLIn that randomly
injects NLs into training data at each training epoch and dynamically increases
the NL injection rate once robust overfitting occurs. Empirically, NoiLIn can
significantly mitigate the AT's undesirable issue of robust overfitting and
even further improve the generalization of the state-of-the-art AT methods.
Philosophically, NoiLIn sheds light on a new perspective of learning with NLs:
NLs should not always be deemed detrimental, and even in the absence of NLs in
the training set, we may consider injecting them deliberately. Codes are
available in https://github.com/zjfheart/NoiLIn.Comment: Accepted at Transactions on Machine Learning Research (TMLR) at June
202
Adversarial Attack and Defense for Non-Parametric Two-Sample Tests
Non-parametric two-sample tests (TSTs) that judge whether two sets of samples
are drawn from the same distribution, have been widely used in the analysis of
critical data. People tend to employ TSTs as trusted basic tools and rarely
have any doubt about their reliability. This paper systematically uncovers the
failure mode of non-parametric TSTs through adversarial attacks and then
proposes corresponding defense strategies. First, we theoretically show that an
adversary can upper-bound the distributional shift which guarantees the
attack's invisibility. Furthermore, we theoretically find that the adversary
can also degrade the lower bound of a TST's test power, which enables us to
iteratively minimize the test criterion in order to search for adversarial
pairs. To enable TST-agnostic attacks, we propose an ensemble attack (EA)
framework that jointly minimizes the different types of test criteria. Second,
to robustify TSTs, we propose a max-min optimization that iteratively generates
adversarial pairs to train the deep kernels. Extensive experiments on both
simulated and real-world datasets validate the adversarial vulnerabilities of
non-parametric TSTs and the effectiveness of our proposed defense. Source code
is available at https://github.com/GodXuxilie/Robust-TST.git.Comment: Accepted by ICML 202
Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection
Adversarial contrastive learning (ACL) does not require expensive data
annotations but outputs a robust representation that withstands adversarial
attacks and also generalizes to a wide range of downstream tasks. However, ACL
needs tremendous running time to generate the adversarial variants of all
training data, which limits its scalability to large datasets. To speed up ACL,
this paper proposes a robustness-aware coreset selection (RCS) method. RCS does
not require label information and searches for an informative subset that
minimizes a representational divergence, which is the distance of the
representation between natural data and their virtual adversarial variants. The
vanilla solution of RCS via traversing all possible subsets is computationally
prohibitive. Therefore, we theoretically transform RCS into a surrogate problem
of submodular maximization, of which the greedy search is an efficient solution
with an optimality guarantee for the original problem. Empirically, our
comprehensive results corroborate that RCS can speed up ACL by a large margin
without significantly hurting the robustness transferability. Notably, to the
best of our knowledge, we are the first to conduct ACL efficiently on the
large-scale ImageNet-1K dataset to obtain an effective robust representation
via RCS